Minimum Description Length Recurrent Neural Networks

Abstract We train neural networks to optimize a Minimum Description Length score, that is, balance between the complexity of network and its accuracy at task. show optimizing this objective function master tasks involving memory challenges go beyond context-free languages. These learners languages such as anbn, anbncn, anb2n, anbmcn +m, they perform addition. Moreover, often do so with 100% accuracy. The are small, their inner workings transparent. thus provide formal proofs perfect holds not only on given test set, but for any input sequence. To our knowledge, no other connectionist model has been shown capture underlying grammars these in full generality.